Expand description
Syn is a parsing library for parsing a stream of Rust tokens into a syntax tree of Rust source code.
Currently this library is geared toward use in Rust procedural macros, but contains some APIs that may be useful more generally.
-
Data structures — Syn provides a complete syntax tree that can represent any valid Rust source code. The syntax tree is rooted at
syn::File
which represents a full source file, but there are other entry points that may be useful to procedural macros includingsyn::Item
,syn::Expr
andsyn::Type
. -
Derives — Of particular interest to derive macros is
syn::DeriveInput
which is any of the three legal input items to a derive macro. An example below shows using this type in a library that can derive implementations of a user-defined trait. -
Parsing — Parsing in Syn is built around parser functions with the signature
fn(ParseStream) -> Result<T>
. Every syntax tree node defined by Syn is individually parsable and may be used as a building block for custom syntaxes, or you may dream up your own brand new syntax without involving any of our syntax tree types. -
Location information — Every token parsed by Syn is associated with a
Span
that tracks line and column information back to the source of that token. These spans allow a procedural macro to display detailed error messages pointing to all the right places in the user’s code. There is an example of this below. -
Feature flags — Functionality is aggressively feature gated so your procedural macros enable only what they need, and do not pay in compile time for all the rest.
Example of a derive macro
The canonical derive macro using Syn looks like this. We write an ordinary
Rust function tagged with a proc_macro_derive
attribute and the name of
the trait we are deriving. Any time that derive appears in the user’s code,
the Rust compiler passes their data structure as tokens into our macro. We
get to execute arbitrary Rust code to figure out what to do with those
tokens, then hand some tokens back to the compiler to compile into the
user’s crate.
[dependencies]
syn = "1.0"
quote = "1.0"
[lib]
proc-macro = true
use proc_macro::TokenStream;
use quote::quote;
use syn::{parse_macro_input, DeriveInput};
#[proc_macro_derive(MyMacro)]
pub fn my_macro(input: TokenStream) -> TokenStream {
// Parse the input tokens into a syntax tree
let input = parse_macro_input!(input as DeriveInput);
// Build the output, possibly using quasi-quotation
let expanded = quote! {
// ...
};
// Hand the output tokens back to the compiler
TokenStream::from(expanded)
}
The heapsize
example directory shows a complete working implementation
of a derive macro. It works on any Rust compiler 1.31+. The example derives
a HeapSize
trait which computes an estimate of the amount of heap memory
owned by a value.
pub trait HeapSize {
/// Total number of bytes of heap memory owned by `self`.
fn heap_size_of_children(&self) -> usize;
}
The derive macro allows users to write #[derive(HeapSize)]
on data
structures in their program.
#[derive(HeapSize)]
struct Demo<'a, T: ?Sized> {
a: Box<T>,
b: u8,
c: &'a str,
d: String,
}
Spans and error reporting
The token-based procedural macro API provides great control over where the
compiler’s error messages are displayed in user code. Consider the error the
user sees if one of their field types does not implement HeapSize
.
#[derive(HeapSize)]
struct Broken {
ok: String,
bad: std::thread::Thread,
}
By tracking span information all the way through the expansion of a
procedural macro as shown in the heapsize
example, token-based macros in
Syn are able to trigger errors that directly pinpoint the source of the
problem.
error[E0277]: the trait bound `std::thread::Thread: HeapSize` is not satisfied
--> src/main.rs:7:5
|
7 | bad: std::thread::Thread,
| ^^^^^^^^^^^^^^^^^^^^^^^^ the trait `HeapSize` is not implemented for `Thread`
Parsing a custom syntax
The lazy-static
example directory shows the implementation of a
functionlike!(...)
procedural macro in which the input tokens are parsed
using Syn’s parsing API.
The example reimplements the popular lazy_static
crate from crates.io as a
procedural macro.
lazy_static! {
static ref USERNAME: Regex = Regex::new("^[a-z0-9_-]{3,16}$").unwrap();
}
The implementation shows how to trigger custom warnings and error messages on the macro input.
warning: come on, pick a more creative name
--> src/main.rs:10:16
|
10 | static ref FOO: String = "lazy_static".to_owned();
| ^^^
Testing
When testing macros, we often care not just that the macro can be used
successfully but also that when the macro is provided with invalid input it
produces maximally helpful error messages. Consider using the trybuild
crate to write tests for errors that are emitted by your macro or errors
detected by the Rust compiler in the expanded code following misuse of the
macro. Such tests help avoid regressions from later refactors that
mistakenly make an error no longer trigger or be less helpful than it used
to be.
Debugging
When developing a procedural macro it can be helpful to look at what the
generated code looks like. Use cargo rustc -- -Zunstable-options --pretty=expanded
or the cargo expand
subcommand.
To show the expanded code for some crate that uses your procedural macro,
run cargo expand
from that crate. To show the expanded code for one of
your own test cases, run cargo expand --test the_test_case
where the last
argument is the name of the test file without the .rs
extension.
This write-up by Brandon W Maister discusses debugging in more detail: Debugging Rust’s new Custom Derive system.
Optional features
Syn puts a lot of functionality behind optional features in order to optimize compile time for the most common use cases. The following features are available.
derive
(enabled by default) — Data structures for representing the possible input to a derive macro, including structs and enums and types.full
— Data structures for representing the syntax tree of all valid Rust source code, including items and expressions.parsing
(enabled by default) — Ability to parse input tokens into a syntax tree node of a chosen type.printing
(enabled by default) — Ability to print a syntax tree node as tokens of Rust source code.visit
— Trait for traversing a syntax tree.visit-mut
— Trait for traversing and mutating in place a syntax tree.fold
— Trait for transforming an owned syntax tree.clone-impls
(enabled by default) — Clone impls for all syntax tree types.extra-traits
— Debug, Eq, PartialEq, Hash impls for all syntax tree types.proc-macro
(enabled by default) — Runtime dependency on the dynamic library libproc_macro from rustc toolchain.
Re-exports
pub use crate::ident::Ident;
Modules
- buffer
parsing
A stably addressed token buffer supporting efficient traversal based on a cheaply copyable cursor. - ext
parsing
Extension traits to provide parsing methods on foreign types. - fold
fold
Syntax tree traversal to transform the nodes of an owned syntax tree. - parse
parsing
Parsing interface for parsing a token stream into a syntax tree node. - A punctuated sequence of syntax tree nodes separated by punctuation.
- spanned
parsing
andprinting
A trait that can provide theSpan
of the complete contents of a syntax tree node. - Tokens representing Rust punctuation, keywords, and delimiters.
- visit
visit
Syntax tree traversal to walk a shared borrow of a syntax tree. - visit_mut
visit-mut
Syntax tree traversal to mutate an exclusive borrow of a syntax tree in place.
Macros
- A type-macro that expands to the name of the Rust type representation of a given token.
- braced
parsing
Parse a set of curly braces and expose their content to subsequent parsers. - bracketed
parsing
Parse a set of square brackets and expose their content to subsequent parsers. - Define a type that supports parsing and printing a given identifier as if it were a keyword.
- Define a type that supports parsing and printing a multi-character symbol as if it were a punctuation token.
- parenthesized
parsing
Parse a set of parentheses and expose their content to subsequent parsers. - parse_macro_input
parsing
andproc-macro
Parse the input TokenStream of a macro, triggering a compile error if the tokens fail to parse. - parse_quote
parsing
andprinting
Quasi-quotation macro that accepts input like thequote!
macro but uses type inference to figure out a return type for those tokens. - parse_quote_spanned
parsing
andprinting
This macro isparse_quote!
+quote_spanned!
.
Structs
- Abi
full
orderive
The binary interface of a function:extern "C"
. - AngleBracketedGenericArguments
full
orderive
Angle bracketed arguments of a path segment: the<K, V>
inHashMap<K, V>
. - Arm
full
One arm of amatch
expression:0...10 => { return true; }
. - Attribute
full
orderive
An attribute like#[repr(transparent)]
. - BareFnArg
full
orderive
An argument in a function type: theusize
infn(usize) -> bool
. - Binding
full
orderive
A binding (equality constraint) on an associated type:Item = u8
. - Block
full
A braced block containing Rust statements. - BoundLifetimes
full
orderive
A set of bound lifetimes:for<'a, 'b, 'c>
. - ConstParam
full
orderive
A const generic parameter:const LENGTH: usize
. - Constraint
full
orderive
An associated type bound:Iterator<Item: Display>
. - DataEnum
derive
An enum input to aproc_macro_derive
macro. - DataStruct
derive
A struct input to aproc_macro_derive
macro. - DataUnion
derive
An untagged union input to aproc_macro_derive
macro. - DeriveInput
derive
Data structure sent to aproc_macro_derive
macro. - Error returned when a Syn parser cannot parse the input tokens.
- ExprArray
full
A slice literal expression:[a, b, c, d]
. - ExprAssign
full
An assignment expression:a = compute()
. - ExprAssignOp
full
A compound assignment expression:counter += 1
. - ExprAsync
full
An async block:async { ... }
. - ExprAwait
full
An await expression:fut.await
. - ExprBinary
full
orderive
A binary operation:a + b
,a * b
. - ExprBlock
full
A blocked scope:{ ... }
. - ExprBox
full
A box expression:box f
. - ExprBreak
full
Abreak
, with an optional label to break and an optional expression. - ExprCall
full
orderive
A function call expression:invoke(a, b)
. - ExprCast
full
orderive
A cast expression:foo as f64
. - ExprClosure
full
A closure expression:|a, b| a + b
. - ExprContinue
full
Acontinue
, with an optional label. - ExprField
full
orderive
Access of a named struct field (obj.k
) or unnamed tuple struct field (obj.0
). - ExprForLoop
full
A for loop:for pat in expr { ... }
. - ExprGroup
full
An expression contained within invisible delimiters. - ExprIf
full
Anif
expression with an optionalelse
block:if expr { ... } else { ... }
. - ExprIndex
full
orderive
A square bracketed indexing expression:vector[2]
. - ExprLet
full
Alet
guard:let Some(x) = opt
. - ExprLit
full
orderive
A literal in place of an expression:1
,"foo"
. - ExprLoop
full
Conditionless loop:loop { ... }
. - ExprMacro
full
A macro invocation expression:format!("{}", q)
. - ExprMatch
full
Amatch
expression:match n { Some(n) => {}, None => {} }
. - ExprMethodCall
full
A method call expression:x.foo::<T>(a, b)
. - ExprParen
full
orderive
A parenthesized expression:(a + b)
. - ExprPath
full
orderive
A path likestd::mem::replace
possibly containing generic parameters and a qualified self-type. - ExprRange
full
A range expression:1..2
,1..
,..2
,1..=2
,..=2
. - ExprReference
full
A referencing operation:&a
or&mut a
. - ExprRepeat
full
An array literal constructed from one repeated element:[0u8; N]
. - ExprReturn
full
Areturn
, with an optional value to be returned. - ExprStruct
full
A struct literal expression:Point { x: 1, y: 1 }
. - ExprTry
full
A try-expression:expr?
. - ExprTryBlock
full
A try block:try { ... }
. - ExprTuple
full
A tuple expression:(a, b, c, d)
. - ExprType
full
A type ascription expression:foo: f64
. - ExprUnary
full
orderive
A unary operation:!x
,*x
. - ExprUnsafe
full
An unsafe block:unsafe { ... }
. - ExprWhile
full
A while loop:while expr { ... }
. - ExprYield
full
A yield expression:yield expr
. - Field
full
orderive
A field of a struct or enum variant. - FieldPat
full
A single field in a struct pattern. - FieldValue
full
A field-value pair in a struct literal. - FieldsNamed
full
orderive
Named fields of a struct or struct variant such asPoint { x: f64, y: f64 }
. - FieldsUnnamed
full
orderive
Unnamed fields of a tuple struct or tuple variant such asSome(T)
. - File
full
A complete file of Rust source code. - ForeignItemFn
full
A foreign function in anextern
block. - ForeignItemMacro
full
A macro invocation within an extern block. - A foreign static item in an
extern
block:static ext: u8
. - ForeignItemType
full
A foreign type in anextern
block:type void
. - Generics
full
orderive
Lifetimes and type parameters attached to a declaration of a function, enum, trait, etc. - A word of Rust code, which may be a keyword or legal variable name.
- ImplGenerics(
full
orderive
) andprinting
Returned byGenerics::split_for_impl
. - ImplItemConst
full
An associated constant within an impl block. - ImplItemMacro
full
A macro invocation within an impl block. - ImplItemMethod
full
A method within an impl block. - ImplItemType
full
An associated type within an impl block. - Index
full
orderive
The index of an unnamed tuple struct field. - ItemConst
full
A constant item:const MAX: u16 = 65535
. - ItemEnum
full
An enum definition:enum Foo<A, B> { A(A), B(B) }
. - ItemExternCrate
full
Anextern crate
item:extern crate serde
. - ItemFn
full
A free-standing function:fn process(n: usize) -> Result<()> { ... }
. - ItemForeignMod
full
A block of foreign items:extern "C" { ... }
. - ItemImpl
full
An impl block providing trait or associated items:impl<A> Trait for Data<A> { ... }
. - ItemMacro
full
A macro invocation, which includesmacro_rules!
definitions. - ItemMacro2
full
A 2.0-style declarative macro introduced by themacro
keyword. - ItemMod
full
A module or module declaration:mod m
ormod m { ... }
. - ItemStatic
full
A static item:static BIKE: Shed = Shed(42)
. - ItemStruct
full
A struct definition:struct Foo<A> { x: A }
. - ItemTrait
full
A trait definition:pub trait Iterator { ... }
. - ItemTraitAlias
full
A trait alias:pub trait SharableIterator = Iterator + Sync
. - ItemType
full
A type alias:type Result<T> = std::result::Result<T, MyError>
. - ItemUnion
full
A union definition:union Foo<A, B> { x: A, y: B }
. - ItemUse
full
A use declaration:use std::collections::HashMap
. - Label
full
A lifetime labeling afor
,while
, orloop
. - A Rust lifetime:
'a
. - LifetimeDef
full
orderive
A lifetime definition:'a: 'b + 'c + 'd
. - A boolean literal:
true
orfalse
. - A byte literal:
b'f'
. - A byte string literal:
b"foo"
. - A character literal:
'a'
. - A floating point literal:
1f64
or1.0e10f64
. - An integer literal:
1
or1u16
. - A UTF-8 string literal:
"foo"
. - Local
full
A locallet
binding:let x: u64 = s.parse()?
. - Macro
full
orderive
A macro invocation:println!("{}", mac)
. - MetaList
full
orderive
A structured list within an attribute, likederive(Copy, Clone)
. - MetaNameValue
full
orderive
A name-value pair within an attribute, likefeature = "nightly"
. - MethodTurbofish
full
The::<>
explicit type parameters passed to a method call:parse::<u64>()
. - ParenthesizedGenericArguments
full
orderive
Arguments of a function path segment: the(A, B) -> C
inFn(A,B) -> C
. - PatBox
full
A box pattern:box v
. - PatIdent
full
A pattern that binds a new variable:ref mut binding @ SUBPATTERN
. - PatLit
full
A literal pattern:0
. - PatMacro
full
A macro in pattern position. - PatOr
full
A pattern that matches any one of a set of cases. - PatPath
full
A path pattern likeColor::Red
, optionally qualified with a self-type. - PatRange
full
A range pattern:1..=2
. - PatReference
full
A reference pattern:&mut var
. - PatRest
full
The dots in a tuple or slice pattern:[0, 1, ..]
- PatSlice
full
A dynamically sized slice pattern:[a, b, ref i @ .., y, z]
. - PatStruct
full
A struct or struct variant pattern:Variant { x, y, .. }
. - PatTuple
full
A tuple pattern:(a, b)
. - PatTupleStruct
full
A tuple struct or tuple variant pattern:Variant(x, y, .., z)
. - PatType
full
A type ascription pattern:foo: f64
. - PatWild
full
A pattern that matches any value:_
. - Path
full
orderive
A path at which a named item is exported (e.g.std::collections::HashMap
). - PathSegment
full
orderive
A segment of a path together with any path arguments on that segment. - PredicateEq
full
orderive
An equality predicate in awhere
clause (unsupported). - PredicateLifetime
full
orderive
A lifetime predicate in awhere
clause:'a: 'b + 'c
. - PredicateType
full
orderive
A type predicate in awhere
clause:for<'c> Foo<'c>: Trait<'c>
. - QSelf
full
orderive
The explicit Self type in a qualified path: theT
in<T as Display>::fmt
. - Receiver
full
Theself
argument of an associated method, whether taken by value or by reference. - Signature
full
A function signature in a trait or implementation:unsafe fn initialize(&self)
. - TraitBound
full
orderive
A trait used as a bound on a type parameter. - TraitItemConst
full
An associated constant within the definition of a trait. - TraitItemMacro
full
A macro invocation within the definition of a trait. - TraitItemMethod
full
A trait method within the definition of a trait. - TraitItemType
full
An associated type within the definition of a trait. - Turbofish(
full
orderive
) andprinting
Returned byTypeGenerics::as_turbofish
. - TypeArray
full
orderive
A fixed size array type:[T; n]
. - TypeBareFn
full
orderive
A bare function type:fn(usize) -> bool
. - TypeGenerics(
full
orderive
) andprinting
Returned byGenerics::split_for_impl
. - TypeGroup
full
orderive
A type contained within invisible delimiters. - TypeImplTrait
full
orderive
Animpl Bound1 + Bound2 + Bound3
type whereBound
is a trait or a lifetime. - TypeInfer
full
orderive
Indication that a type should be inferred by the compiler:_
. - TypeMacro
full
orderive
A macro in the type position. - TypeNever
full
orderive
The never type:!
. - TypeParam
full
orderive
A generic type parameter:T: Into<String>
. - TypeParen
full
orderive
A parenthesized type equivalent to the inner type. - TypePath
full
orderive
A path likestd::slice::Iter
, optionally qualified with a self-type as in<Vec<T> as SomeTrait>::Associated
. - TypePtr
full
orderive
A raw pointer type:*const T
or*mut T
. - TypeReference
full
orderive
A reference type:&'a T
or&'a mut T
. - TypeSlice
full
orderive
A dynamically sized slice type:[T]
. - TypeTraitObject
full
orderive
A trait object typedyn Bound1 + Bound2 + Bound3
whereBound
is a trait or a lifetime. - TypeTuple
full
orderive
A tuple type:(A, B, C, String)
. - UseGlob
full
A glob import in ause
item:*
. - UseGroup
full
A braced group of imports in ause
item:{A, B, C}
. - UseName
full
An identifier imported by ause
item:HashMap
. - UsePath
full
A path prefix of imports in ause
item:std::...
. - UseRename
full
An renamed identifier imported by ause
item:HashMap as Map
. - Variadic
full
orderive
The variadic argument of a foreign function. - Variant
full
orderive
An enum variant. - VisCrate
full
orderive
A crate-level visibility:crate
. - VisPublic
full
orderive
A public visibility level:pub
. - VisRestricted
full
orderive
A visibility level restricted to some path:pub(self)
orpub(super)
orpub(crate)
orpub(in some::module)
. - WhereClause
full
orderive
Awhere
clause in a definition:where T: Deserialize<'de>, D: 'static
.
Enums
- AttrStyle
full
orderive
Distinguishes between attributes that decorate an item and attributes that are contained within an item. - BinOp
full
orderive
A binary operator:+
,+=
,&
. - Data
derive
The storage of a struct, enum or union data structure. - Expr
full
orderive
A Rust expression. - Fields
full
orderive
Data stored within an enum variant or struct. - FnArg
full
An argument in a function signature: then: usize
infn f(n: usize)
. - ForeignItem
full
An item within anextern
block. - GenericArgument
full
orderive
An individual generic argument, like'a
,T
, orItem = T
. - An individual generic argument to a method, like
T
. - GenericParam
full
orderive
A generic type parameter, lifetime, or const generic:T: Into<String>
,'a: 'b
,const LEN: usize
. - ImplItem
full
An item within an impl block. - Item
full
Things that can appear directly inside of a module or scope. - A Rust literal such as a string or integer or boolean.
- MacroDelimiter
full
orderive
A grouping token that surrounds a macro body:m!(...)
orm!{...}
orm![...]
. - Member
full
orderive
A struct or tuple struct field accessed in a struct literal or field expression. - Meta
full
orderive
Content of a compile-time structured attribute. - NestedMeta
full
orderive
Element of a compile-time attribute list. - Pat
full
A pattern in a local binding, function signature, match expression, or various other places. - PathArguments
full
orderive
Angle bracketed or parenthesized arguments of a path segment. - RangeLimits
full
Limit types of a range, inclusive or exclusive. - ReturnType
full
orderive
Return type of a function signature. - Stmt
full
A statement, usually ending in a semicolon. - The style of a string literal, either plain quoted or a raw string like
r##"data"##
. - TraitBoundModifier
full
orderive
A modifier on a trait bound, currently only used for the?
in?Sized
. - TraitItem
full
An item declaration within the definition of a trait. - Type
full
orderive
The possible types that a Rust value could have. - TypeParamBound
full
orderive
A trait or lifetime used as a bound on a type parameter. - UnOp
full
orderive
A unary operator:*
,!
,-
. - UseTree
full
A suffix of an import tree in ause
item:Type as Renamed
or*
. - Visibility
full
orderive
The visibility level of an item: inherited orpub
orpub(restricted)
. - WherePredicate
full
orderive
A single predicate in awhere
clause:T: Deserialize<'de>
.
Functions
- parse
parsing
andproc-macro
Parse tokens of source code into the chosen syntax tree node. - parse2
parsing
Parse a proc-macro2 token stream into the chosen syntax tree node. - parse_file
parsing
andfull
Parse the content of a file of Rust code. - parse_str
parsing
Parse a string of Rust code into the chosen syntax tree node.
Type Definitions
- AttributeArgs
full
orderive
Conventional argument type associated with an invocation of an attribute macro. - The result of a Syn parser.